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Abstract Body 



Background / Context: 

Description of prior research and its inteliectual context. 

Since its inception in the mid-1990s, KIPP has grown from two core middle schools to a nationwide 
network of 109 charter schools in 20 states and the District of Columbia as of the 2011-2012 school 
year. KIPP now serves over 32,000 students at the elementary, middle, and high school levels and plans 
to open approximately 50 new schools over the next three years. In 2010, the KIPP Foundation was 
awarded a five-year, $50 million Investing in Innovation (13) scale-up grant by the U.S. Department of 
Education's Office of Innovation and Improvement. The award was one of four 13 scale-up grants 
funding expansion of programs demonstrating strong evidence of previous effectiveness in improving 
student achievement and educational attainment.* The KIPP Foundation plans to use the 13 grant to 
scale up its program while sustaining KIPP's positive impacts. 

Though the average effects of charter schools overall are mixed, the consistently positive effects of KIPP 
charter schools suggest that their model may be worth replicating. Recent studies of KIPP schools in the 
San Francisco Bay Area and Lynn, Massachusetts— using propensity score matching and a lottery-based 
design, respectively— found large and statistically significant impacts in both math and reading, with 
effect sizes ranging from 0.12 to 0.68 standard deviation units in reading and from 0.19 to 0.88 standard 
deviation units in math for one year of KIPP instruction (Woodworth et al. 2008; Angrist et al. 2010). A 
2010 Mathematica study of 22 KIPP middle schools estimated impacts after three years (Tuttle et al. 
2010). By Year 3, half of the KIPP schools in the sample produced math impacts of 0.48 standard 
deviations or more (representing an estimated 1.2 years of additional instruction), and half of the KIPP 
schools in the sample produced three-year reading effects of 0.28 standard deviations or more 
(representing an estimated 0.9 year of additional instruction). 

Flowever, to date, no rigorous research has been published on KIPP schools at the elementary or high 
school levels, which represent the majority (41 and 27 percent, respectively) of new school openings 
over the past five years, nor have any existing studies compared impacts over time as the network has 
grown. 



Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

As a condition of its 13 grant, KIPP contracted with an independent evaluator (Mathematica) to address a 
key research question: does KIPP maintain its demonstrated effectiveness as it scales? While this 
question sounds simple enough in theory, it poses several methodological and practical challenges. This 
paper outlines some of those key challenges and presents potential approaches for addressing them. 

Our ongoing work examines variation in impacts across KIPP middle schools. To some extent, many 
"typical" sources of variation across schools aren't applicable in the case of KIPP due to its consistent 



http://www.ed.gov/news/press-releases/nations-boldest-education-reform-plans-receive-federal- 
innovation-grants-once-pr, accessed Aprii 12, 2011. 
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model/ But even that consistent model leaves two key dimensions of potential variation: variation in 
impacts across the different grade spans of KIPP schools, and variation in impacts between previously- 
established and newly-opened KIPP schools as the network grows in the number of schools and students 
it serves. 

First, KIPP is intentionally expanding vertically— into elementary and high schools in cities and locations 
where KIPP middle schools are already established. Yet the evidence on KIPP is entirely focused on 
middle schools. Will the positive impacts found in past studies of KIPP persist to the same extent in 
elementary and high school? There are reasons to believe that they might not. At the elementary school 
level, unlike the middle school level, KIPP schools are not enrolling kids who have fallen behind in school 
for five or six years already. At the high school level, the issue is somewhat different: KIPP is educating 
students who most often have already been in KIPP (attended a KIPP middle school) for up to four years. 
In this way, there may not be sufficient "room" to generate additional impacts for these students. 

Second, KIPP is also expanding "horizontally"— in some cases into new cities, but primarily by adding 
new middle schools in existing markets. In this way, the "treatment" at a new KIPP school may not be 
the same at newly established schools as at the early (or existing) KIPP schools, since it may be harder to 
recruit teachers and school leaders, attract students, and so forth. On the other hand, schools in 
existing markets may be able to capitalize on economies of scale from regional structures and supports 
already in place. 

Generally, our plans for estimating impacts include a combination of approaches. We will evaluate 
KIPP'S impacts on student achievement by capitalizing on the advantages of both experimental and 
quasi-experimental designs (QEDs). An experimental approach, or randomized control trial (RCT), can 
provide the most rigorous assessment of impacts on student outcomes, but it may be applied only in 
oversubscribed schools (those with more applicants than available slots) that hold lotteries to determine 
admission. The QED may be less rigorous given the potential for selection bias, but it offers the 
advantage of applicability to most KIPP schools in that oversubscription is not requisite for comparison 
groups to be identified from similar students attending nearby schools. 

To complicate matters further, neither of these approaches can practically be applied to each of the 
three grade spans of schools (elementary, middle, and high; see Figure 1). As it currently stands, 
elementary and middle school entry grades both represent major points of entry into the KIPP program, 
and oversubscription at those grades is fairly common. To date, KIPP high schools have focused on 
serving students who have attended KIPP middle schools, resulting in a pool of outside applicants 
potentially participating in admissions lotteries too small to support an RCT. Conversely, we must 
exclude any KIPP elementary schools from any QED approaches, since we lack pre-test data for their 
students (who enroll as young as three years of age in some cases)— the key variable for generating a 
credible comparison group. 

Given these considerations, our estimation strategy will involve three different types of designs: 

1. RCT. A lottery-based RCT will compare lottery winners (treatment group) to lottery losers 
(control group) at KIPP elementary and middle schools. 



^ Nevertheless, we find variation in impacts across different KIPP middle schools, ranging from -0.12 to +0.76 
standard deviation units in math and -0.14 to +0.43 in reading after one year. Potential factors related to this 
variation in impacts will be examined in a forthcoming report expected in 2012. 
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2. Matched-student QED. A student-level propensity-score-matched comparison group QED will 
compare students at KIPP middle and high schools (treatment group) to similar non-KIPP 
students (comparison group). 

3. Matched-school QED. A student-level QED derived from school-level matching will compare 
outcomes for KIPP middle school students with and without the opportunity to later attend a 
KIPP high school (comprising the treatment and comparison groups, respectively). One model 
will compare students within a region, over time (before and after the KIPP high school opens); 
another will compare a single cohort of students across regions (with and without KIPP high 
schools). 

Using these models, we will attempt to examine three main sources of variation in the impacts of 
KIPP schools on student achievement: 

1. Are newly opened KIPP schools in areas already saturated with other KIPP schools less effective 
than the previously opened schools? 

2. Are the impacts of KIPP schools related to school characteristics that may be associated with the 
growth of the network— such as key characteristics of principals or students? 

3. Is the KIPP model, shown to be successful at the middle school level, as successful at the 
elementary and high school levels? 
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Appendix B. Tables and Figures 



Figure 1. Eligibility of Schools for Impact Analysis 
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